Interpretation of hybrid generative/discriminative algorithms

In discriminant analysis, probabilistic generative and discriminative approaches represent two paradigms of statistical modelling and learning. In order to exploit the best of both worlds, hybrid modelling and learning techniques have attracted much research interest recently, one example being the so-called hybrid generative/discriminative algorithm proposed in Raina et al. (2003) and its multi-class extension (Fujino et al., 2007). In this paper, we interpret this hybrid algorithm from three perspectives, namely class-conditional probabilities, class-posterior probabilities and loss functions underlying the model. We suggest that the hybrid algorithm is by nature a generative model with its parameters learnt through both generative and discriminative approaches, in the sense that in fact it assumes a scaled data-generation process and uses scaled class-posterior probabilities to perform discrimination. Our suggestion can also be applied to the multi-class extension. In addition, using simulated data, we compare the performance of the normalised hybrid algorithm as a classifier with those of the näıve Bayes classifier and linear logistic regression. In general, our simulation studies suggest the following: if the covariance matrices are diagonal matrices, the näıve Bayes classifier performs the best; if the covariance matrices are non-diagonal matrices, linear logistic regression performs the best. In other words, our studies cannot support that the hybrid algorithm